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ABSTRACT 

There are issues in evaluation and stages of product 
development that demand the use of experimental or quasi-experimental 
designs. To counteract criticism of evaluation efforts, an approach 
to the examination of the multiple issues involved in curriculum 
product evaluation across the usual developmental cycle of 
educational products was developed. Curriculum products typically 
move through a developmental sequence comprised of five stages: 
Initial state. Hot House - the initial tryout of a prototype product, 
pilot test, field test, and public diffusion. Each stage represents a 
milestone in the life of a product. In the course of evaluation, 
various audiences are acquired: the sponsor, the institution, the 
developer, consumer representatives, and advisors. There are five 
major dimensions of a comprehensive evaluation of curriculum 
products: Desirability/Feasibility, (2) Management/Procedural Cost, 
Product Worth, Usability, and Generalizability. Issues relating to 
the continuation or termination of a program concern statement and 
fulfillment of objectives, establishing a rationale for the use of 
particular measuring instruments, determination of whether or not 
different effects result from alternative procedures. When the 
product enters the diffusion stage, formative evaluation is ended and 
summative evaluation ought to begin. It is pointed out that true 
summative evaluation is consumer protection and is a three tiered 
process operating wherein: (1) The product developer establishes the 

criteria; (2) some agency of the federal government examines the 
product; and (3) Local education agencies research the products. 
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I PRODUCTION 

The recent past has witnessed the emergence of curriculum evaluation 
as a critical concern among the educational research and other communities. 
The major impetus for this concern has come from various curriculum proj- 
ects such as that of the Physical Science Study Committee and the funding 
of laboratories and centers for curriculum research and development by the 
U. S. Office of Education. Models for evaluation activities in connection 
with curriculum development activities have been proposed by scholars in 
various disciplines, and we now appear to be at that point in time where 
it becomes possible to attempt a synthesis of some of these ideas into a 
coherent system for the evaluation of curriculum products. 

Evaluation, as the most salient term in this paper, requires defini- 
tion. The recent literature (especially Guba and Stuff lebeam [1968], 

Stuff lebeam [1971], MacDonald 11970], Astin and panes H97U) has tended to 
foster a view of curriculum evaluation tn terms of its role in the realm of 
educational decision-making, in this role, curriculum evaluation may be 
defined in the manner of Guba and Stuff lebeam (1971) as "The process of 
obtaining and providing useful information for making educational decisions 
(pp. 23-24)." The critical terms are "useful" and "dectston." We would 
argue (with Guba and Stufflebeam [1968], Asttn and Panes [1971], and others] 
that unless the information gathered and reported will be used tn the 
determination of which among alternative options will be chosen, then there 
Is little or no reason to engage in evaluation. 
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The issue that has perhaps been most pervasive of late In '.he field 
of curriculum evaluation was raised most succinctly in Scriven's (1968) 

AERA monograph paper. In dealing with the roles and goals of evaluation, 
Scriven posited the distinction between formative and summative evaluation. 
Briefly, formative evaluation refers to those activities that provide 
Information to the developer concerning his product's worth and effective- 
ness when the product is still fluid and most amenable to change; summa- 
tive evaluation takes place when a product is a completed entity. 

The recent literature has dealt extensively with the formative- 
summative dichotomy (e.g., Grobman, 1968; Cronbach, 1963; Karl, 1970; and 
Wittrock, 1966). As Weiss (1971) has noted, however: 

In contrast with the large body of literature concerned 
with measuring and diagnosing student progress is a lack 
of clear cut systematic procedures for the logical and 
judgmental questions of formative curriculum evaluation. (p.*0 

Moreover, some individuals have viewed the distinction as being one of 
rigor rather than one of role. Scriven, himself, refuted that interpre- 
tation when he writes, "If you think that formative evaluation can be 
informal and that only summative needs to be done with any kind of rigor . . . 
then you find yourself simply making the wrong Intermediate decisions 

(1971, p.D." 

Marion Karl (1970), in his explication of process evaluation, (". . . 
evaluation that illuminates areas where revisions are necessary while the 
opportunity for revision still exists"), notes that few attempts are made to 
conduct evaluation of this type and attempts to explain this lack. 
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No neat, scientific research design is probable. Assumptions 
inherent in the models for statistical analysis of the data 
must be violated. The researcher doesn't have a tenth of the 
control he would like. Subjectivity is rampant. At times, 
one feels he is attempting a task as impossible as analyzing 
the water at a given point in a moving stream, (p. 1) 



Indeed, the timely distinction between formative and summative 
evaluation has served to focus attention on the suitability of the 
application of experimental or quasi -experimental designs (Campbell and 
Stanley, 1963) as the sine qua non of evaluation methodology. During 
the course of the development of the product some of the purposes of 
the evaluation activies may not be achievable by the application of 
experimental methodology. As Guba and Stufflebeam (1968) have noted: 

On the surface, the application of experimental design to 
evaluation problems seems reasonable, since traditionally 
both experimental research and evaluation have been used 
to test hypotheses about the effects of treatments. How- 
ever, there are four distinct flaws with this reasoning. 

1. The application of experimental design to evaluation 
problems conflicts with the principal that evaluation 

should facilitate the continual improvement of a program . . . 

2. Experimental design is useful for making decisions 
after a project has run full cycle, but almost useless as 
a device for making decisions during the planning and 
implementation of a project . . . 

3. Experimental design is suited to the antiseptic condi- 
tions of the laboratory but not to the septic conditions 
of the classroom . . . 

While internal validity may be gained through the. 
control of extraneous variables, such an achievement is 
accomplished at the expense of external validity . . . 

(n. 1*4-16) 

As we hope to demonstrate later in this paper, there are issues in evalu- 
ation and stages of product development that demand the use of experimental 
or quasi-experimental designs, but to require that all evaluation activities 
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conform to this single set of standards is to adopt a procrustean solution to 
an array of complex problems . J^Phe re are legitimate questions, critical to 
the development of a curriculum, for which experimental designs are inapplic- 
able. In short, we argue that formative evaluation needs to be formal and 
rigorous, but need not in all instances be equivalent to experimental research. 

There has been some criticism of evaluation efforts based on the grounds 
that they examine only student learning outcomes and usually only outcomes that 
fa 1 1 within a narrow band, i.e., those that can be behavioral ly specified 
a priori. It strikes us that this criticism is just and warranted. It is no 
doubt true that the ultimate worth of most educational products rests largely 
on whether they are effective agents for the promotion of student learning, 
yet there are other important criteria, not only ultimate but intermediate 
and emergent. Considerations of intermediate and emergent issues necessi- 
tate a dynamic view of curriculum evaluation. As Grobman (1968) has stated: 

. . . the curriculum itself is emergent. As curriculum ideas 
work, they are pursued further; as they fail, they are modified 
or scrapped. In the same way, the evaluation is emergent; it 
must adapt to the readily changing experimental curriculum and 
to the changing needs of the project . . . perhaps emergent and 
dynamic are the two best adjectives to describe curriculum 
project evaluation, (p. 10) 

Indeed, It may be argued that it Is incumbent upon the evaluation 
program to provide information beyond whether or not particular learning- 
outcome goals were achieved, since such Information is at best only mini- 
mally sufficient for decisions concerning adoption or rejection by a 
school system or other "purchaser." Again, borrowing from Grobman (1968) 
we may note: 

A curriculum assessment that concerns itself only with the 
instructional materials, without some understanding of the 
other variables in the situation, may conclude that certain 
results are produced, but may give no indication of why these 
are produced, or why different results ensue in different 
situations. Or it may conclude that the materials have 
failed, when the failure reflects a different factor, (p. 5) 
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